Head Start REDI

Study 1

Teachers delivered the 33 lesson Preschool PATHS curriculum once per week to promote children's social-emotional skills. REDI targeted vocabulary, syntax, phonological awareness, and print awareness with three program components. Teachers received weekly mentoring support provided by local education consultants (REDI trainers), experienced master teachers who were supervised by two project-based senior educational trainers. The program was integrated into Head Start centers that were implementing the High/Scope or Creative Curriculum.

Summary

The Head Start REDI study (Bierman, Domitrovich et al., 2008; Bierman, Nix et al. 2008; Bierman et al. 2014; Nix et al., 2013, 2016; Welsh et al., 2020; Bierman et al., 2020) was designed as a randomized trial that involved 25 centers with 44 classrooms. Centers were randomly assigned to either the treatment condition (REDI) or to the usual practice of Head Start. The classrooms were drawn from three counties in Pennsylvania and involved a mix of large and small towns. In total, 356 4-year-olds participated. They were assessed pre-intervention, post-intervention, 1 year after the intervention, and annually up to 10 years after the intervention. Children were observed during playtime and assessed by their parents and teachers.

Bierman, Domitrovich et al. (2008), Bierman, Nix et al. (2008), Bierman et al. (2014), Nix et al. (2013; 2016), Welsh et al. (2020), and Bierman et al. (2020)

• Parents and teachers reported less aggression among those in the intervention group when compared to the control group at posttest and follow-up.
• Students in the intervention group showed more improvements in emergent literacy skills than control students at posttest and, for a measure of phonemic decoding, at follow-up.
• Children in the intervention group showed more improvements on child vocabulary and on parent reports of communication and language use at home at posttest.
• At the 4-year follow-up, intervention subjects showed more positive trajectories for several measures of socio-emotional functioning such as social competence, aggressive-oppositional behavior, and peer rejection.
• At the 6-year follow-up, intervention children, compared to control children, demonstrated significantly better social adjustment.
• At the 8- and 10-year follow-ups, intervention children, compared to control children, demonstrated significantly fewer conduct problems.
• At the 8- and 10-year follow-ups, intervention children, compared to control children, demonstrated significantly fewer emotional symptoms.

Effects on Risk and Protective Factors

• Students in the intervention group showed more improvement than those in the control group on emotional understanding at posttest and social problem-solving skills at posttest and follow-up.
• At the 4-year follow-up, intervention subjects showed more positive trajectories for several measures of socio-emotional functioning such as learning behavior, attention problems, and student-teacher closeness.
• At the 6-year follow-up, intervention children showed significantly higher academic engagement.
• At the 6-year follow-up, intervention children showed significantly higher parent involvement.

Evaluation Methodology

Design: The study consisted of a randomized sample taken of 44 Head Start classrooms in 25 (Nix et al., 2013, 2016) centers across three counties in Pennsylvania. Half of the participating classrooms came from a large, fairly densely populated county in the southeastern part of the state, which included an urban community surrounded by smaller communities. The other classrooms came from two smaller counties in the central part of the state, characterized by small towns and rural areas. Classrooms were stratified on location, length of program, and student demographics to assure even representation in the intervention and comparison conditions. In the more urban county, classrooms were stratified into three groups, in terms of urban (vs. nonurban) location, percentage of minority students served, and use of Spanish in the classroom. In the two more rural counties, classes were also stratified into three groups, which varied in terms of being full-day or half-day programs and located in small towns versus rural locations.

Classrooms in the same center were always assigned to the same condition, to avoid inadvertent contamination of condition within centers. Within stratified groups, centers were randomly assigned to intervention or comparison conditions. This process resulted in 14 classrooms (67%) in each condition that were located in small centers (1-2 classrooms) and 8 classrooms in each condition located in four larger centers (containing 3-5 classrooms), with children from ethnic minority groups fairly evenly spread across condition (39% of the intervention group and 45% of the control group.) A total of 356 children took part in the study.

Children were recruited across two years. Each year, teachers were studied as they implemented the intervention for the first time, and 4-year-old children were assessed after receiving one year of REDI intervention and then one year later at the end of kindergarten. Teachers in the comparison classrooms continued to conduct the curriculum "as usual." These control classrooms contained about 14 children, typically one-third three-year olds and two-thirds four-year olds. To recruit participants for the evaluation trial, brochures describing the study were distributed to parents of all 4-year-old children in participating classrooms. Overall, only 14 eligible families declined to participate in the evaluation process, but an additional 21 primary caregivers were very difficult to reach and failed to complete the preassessment. Two children were dropped because they had a twin or sibling in the study, and 19 families withdrew early from Head Start and completed only parts of the assessment procedures. Hence, study participants represented 86% of the initially eligible population.

Assessments occurred early in the school year (pretest) and at the end of the school year (posttest). At the 1-year follow-up, the subjects transitioned to 202 kindergarten classes at 82 schools in 33 school districts. The follow-up assessment occurred in March-April, near the end of kindergarten. The 4-year follow-up assessed the children each year through the end of third grade. The 6-year follow-up assessed the children during 5^th grade. The 8- and 10-year follow-ups assessed the children during 7^th and 9^th grade, respectively.

Among the 356 children, attrition was only 3% at posttest and only 5% at follow-up. At the 4-year follow-up, attrition reached only 9%. At the 6- and 10-year follow-ups, attrition was 19%.

Sample: Participants in this study included two cohorts of 4-year old children. They were 17% Latino, 25% Black, and 58% white. Also, 54% were girls and 46% were boys. Primary caregivers included 89% mothers, 4% fathers, 4% grandparents, and 3% other, such as stepparents or foster parents. In terms of education of the primary caregivers, 31% had less than a high school education, 60% graduated from high school, 8% completed a technical degree, and 2% completed a college degree. The authors note that the sample is typical of the socioeconomically disadvantaged families and children for whom Head Start is intended.

Measures: A multi-method, multi-measure assessment battery included child assessments, teacher ratings, parent ratings, and direct observations. Outcome measures at posttest represented six core domains: (a) language skills, (b) emergent literacy skills, (c) emotional understanding and social-cognitive skills, (d) social-emotional behaviors, (e) learning engagement at school, and (f) learning engagement at home. Bierman, Domitrovich et al. (2008) said posttest observational measures were obtained from observers naive to condition, and Bierman et al. (2014, p. 156) said with reference to the 1-year follow-up measures that child interviewers and kindergarten teachers did not know intervention group status. However, Head Start teachers both implemented the program and provided some measures at posttest.

Language skills. Three tests were administered directly to children to assess their language skills. In the Expressive One-Word Picture Vocabulary Test, children gave the word that best described pictures they were shown. The Grammatical Understanding subtest of the Test of Language Development assessed syntax comprehension. Children listened to a sentence and chose one of the four pictures that "best matched" the meaning of the sentence. The Sentence Imitation subtest assessed syntax expression. Children repeated sentences read aloud by the interviewer. Scores reflected the number of increasingly complex sentences a child imitated correctly.

Emergent literacy skills. Three subscales assessing emergent literacy skills were drawn from the Test of Preschool Early Literacy (TOPEL). The Blending subtest assessed phonological processing. Children were asked to combine different parts of a word, such as "hot" and "dog" or "b" and "air" and point to the correct picture or say the full word. On the Elision subtest of TOPEL, children deconstructed compound words and pointed to the correct picture. On the Print Knowledge subtest of the TOPEL, children identified pictures of letters or words and named letters.

The 1-year follow-up (Bierman et al., 2014) used somewhat different measures at the end of kindergarten. Instead of the Blending measure, the follow-up used a measure of Phonemic Decoding Efficiency based on the number of words sounded out accurately. Instead of the Elision measure, the follow-up used a measure of Sight Word Efficiency based on the number of words read accurately. Instead of the Print Knowledge measure, the follow-up used a measure of letter-word identification from the Woodcock-Johnson Test of Achievement.

Emotional understanding and social-cognitive skills. Two measures assessed emotional understanding. On the Assessment of Children's Emotion Skills, children determined whether the facial expressions in 12 photographs reflected happy, mad, sad, scared, or no feelings. The score was the total number correctly identified. On the Emotion Recognition Questionnaire children listened to 16 stories describing characters in emotionally evocative situations and identified their feeling by pointing to pictures of happy, mad, sad, or scared faces. Children received a score of 2 for correctly identifying the feeling and a score of 1 for correctly identifying the valence. In addition, social-problem solving skills were assessed using a variation of the Challenging Situations Task. Children were presented with pictures of four peer scenarios (e.g., a peer knocking down blocks, being hit, entering a group, a peer taking a ball). After each scenario, children were asked what they would do in the situation. Their open-ended responses were coded as competent (i.e., appropriately asserting oneself or calmly negotiating a solution), aggressive (i.e., responding with verbal or physical antagonism, intimidation, or force), or inept.

Social-emotional behaviors. Teacher ratings, observer ratings, and parent ratings assessed social competence and aggressive-oppositional behavior. The 13 items of the Social Competence Scale were rated on a 6-point Likert scale (never to almost always) and included prosocial behaviors such as sharing, helping, and understanding other's feelings, as well as self-regulatory behaviors, such as resolving peer problems independently. Ratings provided by lead and assistant teachers were averaged.

Seven items from the Teacher Observation of Child Adaptation-Revised (TOCA-R) assessed overt aggression (e.g., stubborn, yells, fights). Six items from the Preschool Social Behavior Scale-Teacher Form assessed relational aggression (e.g., "Tells other kids he/she won't be their friend unless they do what he/she wants"). Items were rated on a 6-point Likert scale (almost never to almost always). Ratings from lead and assistant teachers were averaged and overt and relational ratings were combined to form a total aggression score. Parents and observers completed the seven items from TOCA-R only.

Learning engagement at school. Teacher ratings were used to assess learning engagement at school, using an eight-item inventory developed for this study. Items were rated on a 6-point Likert scale (strongly disagree to strongly agree), and reflected self-regulation (e.g., "Has the self-control to do well in school" and "Can follow the rules and routines that are part of the school day"), learning motivation and involvement (e.g., "Seems enthusiastic about learning new things"), and compliance (e.g., "Is able and willing to follow teacher directions"). Lead and assistant teacher ratings were averaged.

Observers completed the Adapted Leiter-R Assessor Report to assess task orientation at school. After the child assessments, interviewers rated the child's participation and involvement, using 13 items (e.g., "Shows pleasure in accomplishment and active mastery," "Careful and interested in accuracy," "Alert and interactive"). Each item was rated on a 4-point scale. Scores from the two assessment sessions were averaged.

Teachers also completed the ADHD Rating Scale. Based on the Diagnostic and Statistical Manual symptoms of attention deficit hyperactivity disorder, this scale includes 14 items reflecting attention problems (e.g., "Is easily distracted," "Has trouble following directions"), each rated on a 4-point scale. Lead and assistant teacher ratings were averaged and used in this study to represent a continuous dimension reflecting difficulties with impulse control, distractibility, and sustained attention.

Learning engagement at home. Parents completed the ADHD Rating Scale. In addition, they responded to five questions about children's language and communication at home (e.g., "How many times in a typical week do you and your child have a conversation that lasts 10 minutes or more?", "How often does your child volunteer to tell you about something that happened when you were not with him or her?"). Finally, parents answered six questions about children's engagement in reading at home (e.g., "When was the last time you and your child read a book together?", "How many books did you read at that time?").

Mediation. Bierman, Nix et al. (2008) examined three of the 12 skill measures and four of the 11 behavior measures. Unique to their study, they examined five measures of executive function as outcomes, moderators, and mediators: 1) Backward Word Span measured the ability to repeat a list of words in backward order; 2) Peg Tapping measured the ability of children to tap their peg twice when the interviewer tapped once and vice versa (alpha for 16 trials = .87); 3) Dimensional Change Card Sort (DCCS) measured the ability of children to correctly sort cards by, for example, color after initially sorting them by, for example, size (alpha for six trials = .93); 4) Walk-a-Line Slowly measured the ability of children to walk on a straight line at a slow pace; and 5) Task Orientation measured the ability to focus on a task during a testing session (ratings of multiple dimensions had an alpha of .93).

Follow-up. Outcome measures at 1-year follow-up (Bierman et al., 2014) included 13 items in four domains: a) language and emergent literacy skills, b) learning engagement, c) social competence, and d) aggressive behavior. The follow-up used 10 of the 23 measures used at pretest and posttest. As described above, three other measures of emergent literacy skills were new to the follow-up but were related to the pretest and posttest measures.

Although not treated as outcomes, three measures of kindergarten context served as potential moderators in the follow-up. Observers rated kindergarten classrooms of the study subjects on the quality of student-teacher interactions and the quantity of reading instruction. In addition, standardized tests measured the percent of third-grade scores below the basic category in reading and math proficiency.

In the mediation analysis (Nix et al., 2013), posttest outcome measures served as mediators, and differed in some ways from posttest measures used in other articles. These mediating measures included 1) vocabulary, 2) emergent literacy skills based on a combination of the blending and Elision measures, 3) emotional understanding based on a combination of measures of emotional skills and emotional recognition, 4) competent problem solving, and 5) positive social behavior based on a combination of observer, teacher, and parent ratings of social competence and aggressive behavior (reverse coded).

The mediation analysis used several final outcome measures obtained at the kindergarten follow-up. They included 1) reading achievement based on three measures, 2) learning engagement based on a combination of measures of intellectual curiosity, self-discipline, and attention, and 3) positive social behavior based on the same items as at posttest.

6-year follow-up (Welsh et al., 2020)

At the 6-year follow up, the children's fifth grade teachers completed measures of the children's social adjustment, problem behaviors, academic engagement, and parent involvement. The teachers were unaware of the intervention assignment made in preschool and provided independent ratings.

Social adjustment was assessed using the Social Competence Scale (α = .94 - .96 across data collection waves), four items from the Excluded by Peers subscale of the Child Behavior Scale (α = .83 - .88), and the closeness subscale of the Student-Teacher Relationship Scale (α = .88 - .92) These three scales were averaged to form a composite measure of social adjustment (α = .72 - .77).

Problem behaviors were assessed using 7 items from the Authority Acceptance subscale of the Teacher Observation of Classroom Adaptation - Revised Scale (α = .89 - .93) and 6 items from the Children's Social Behavior Scale - Teacher Version (α = .94 - .96). These two measures were significantly correlated at each time point (r = .50 - .66) and were averaged.

Academic engagement was assessed using the School Readiness Questionnaire (α = .93 - .97) and the Inattentive-Impulsive subscale of the ADHD Rating Scale (α = .93 - .96). These two measures were significantly correlated at each time point (r = .72 - .81) and were averaged.

Parent involvement was measured using the Parent-Teacher Involvement Questionnaire (α = .90 - .95).

8- and 10-year follow-ups (Bierman et al., 2020)

When the children were in 7^th and 9^th grade, their teachers rated them using the Strengths and Difficulties Questionnaire. This scale contained subscales for conduct problems, emotional symptoms, hyperactivity-inattention, and peer problems. Alphas for these measures across the two time points ranged from .64 to .86.

Analyses: To examine preintervention differences between the intervention and the comparison groups, hierarchical linear models (HLM) were estimated, accounting for the nesting of children within classrooms. Child sex and race were included as Level 1 covariates and center site, cohort, and intervention status were included as Level 2 covariates in each of these models.

The first set of analyses to examine postintervention group differences used HLM to examine the 11 measures of child language, emergent literacy, emotional understanding, and social problem-solving skills that were targeted directly by the intervention and were measured using direct assessments of child skill knowledge. Because the intervention was delivered at the classroom level, analyses accounted for the nonindependence of data at that level. Child sex and race were included as Level 1 covariates and site, cohort, and intervention status were included as Level 2 covariates. Preintervention scores were available for all these measures and also were included as Level 1 covariates. For ease of interpretation, all measures were standardized with a mean of 0 and a standard deviation of 1. By doing this, the coefficient for the intervention effect represents the difference in average expected scores between children in the intervention and control groups as a proportion of a standard deviation.

HLM analyses were also conducted to assess intervention effects on the 12 behavioral ratings provided by teachers, observers, and parents. As before, the HLM nested children within classrooms, included child sex and race as Level 1 covariates and included site, cohort, and intervention status as Level 2 covariates. In this case the researchers only had preintervention parent ratings and preintervention observing ratings of task orientation as Level 1 covariates.

The follow-up analysis used cross-classified hierarchical models in which Level-2 random effects accounted for clustering within both preschool classrooms and kindergarten school districts. The Level-1 equation controlled for baseline outcomes (or something similar when measures changed at follow-up). With outcome measures standardized, the coefficients are comparable to Cohen's d.

The mediation analysis used full information maximum likelihood estimation to minimize any bias that comes from missing data. It also used multilevel models that nested children within their Head Start classrooms. The models controlled for baseline values of the mediating variables by treating them as residualized gain scores, but it appears that the models did not use baseline controls for the kindergarten outcomes.

With data on five time points (posttest, kindergarten, first grade, second grade, and third grade), the 4-year follow-up (Nix et al., 2016) compared developmental trajectories across conditions. For each of six outcomes, latent class growth models identified from two to four trajectory patterns and each child was assigned to the trajectory that he or she most likely followed. The test of the intervention effect was based on the significance of differences in the proportion of children in each developmental trajectory who had been in Head Start REDI versus Head Start as usual. Odds ratios for intervention children following a particular developmental trajectory relative to control children showed the magnitude of the program effect. The analysis was done at the individual level without adjustment for clustering within classrooms (though the ICC = .003) or schools.

For the 6-year follow-up outcomes (Welsh et al., 2020), growth curve models were estimated for each outcome. The models allowed for both random slopes and random intercepts for time within individuals but not for preschool centers, the unit of assignment, because clustering within centers was found to be negligible. Mean growth curves were examined for children in the intervention and control groups from posttest through the 6-year follow-up. Initial growth curves allowed for linear, quadratic, and cubic effects of time. In all growth models, all of the interactions of time with intervention status were nonsignificant or failed to improve model fit and were trimmed. Models controlled for study design features (e.g., cohort, county site), demographics (e.g., child sex, age, race, and family income-to-needs ratio), direct assessments of child skills (e.g., IQ, vocabulary, executive functions, emotional understanding). To control for baseline outcomes, models included baseline parent ratings of scales of social adjustment, aggressive problem behaviors, and attention problems. There was not a comparable measure of parent involvement at baseline.

For the 8- and 10-year follow-ups (Bierman et al., 2020), intervention effects were examined using three-level hierarchical linear models. Level 1 included fixed effects for time (7^th or 9^th grade assessments), Level 2 included fixed effects for intervention status and covariates (study design features, demographics, baseline assessments of child skills, and similar though not identical baseline measures of outcomes), and Level 3 nested participants within their Head Start classroom. All analyses were conducted with 40 imputed datasets that imputed missing covariate and outcome data.

Outcomes

Implementation Fidelity: Teachers reported completing most lessons, and trainers rated the average fidelity and quality of implementation as between adequate and strong.

Baseline Equivalence: No significant pretreatment intervention-control group differences emerged on the 11 measures of child skills or the 6 measures of child behavior for which pretreatment scores were available. The study did not report on equivalence of sociodemographic characteristics.

At follow-up (Bierman et al., 2014), intervention and control subjects in kindergarten were compared on measures of teacher-student interaction quality, curriculum emphasis on reading instruction, and school-level achievement. The conditions did not differ significantly on these three measures, thus confirming that on entering kindergarten, intervention students were not selected into better schools. Welsh et al. (2020) reported that there were no significant differences at baseline on a number of study design features, demographics, and assessments of child skills. Bierman et al. (2020) reported no significant differences on study design features, demographics, or outcomes at baseline.

Attrition: Attrition was low - only 3-5%. But Nix et al. (2013, p. 1003) stated that there were "few significant differences" between those retained and not retained. Nix et al. (2016) presented no information on differential attrition for the 9% lost by the 4-year follow-up. Welsh et al. (2020) reported 19% attrition at the 6-year follow up. They reported one significant predictor of attrition: African American participants were more likely to drop out. However, their rates of attrition did not differ by condition. Bierman et al. (2020) reported a 19% attrition rate at 10-year follow up. Retention was significantly higher in the control group (88%) compared to the intervention group (78%) and among White non-Latino children (87%) compared to Latino children (72%).

Posttest: In Bierman, Domitrovich et al. (2008), there were significant treatment effects for 7 of 11 skills directly targeted by the intervention and marginally significant intervention effects for two others. Three of 12 behavioral ratings provided by teachers, observers, and parents significantly favored intervention students and five measures showed marginally significant trends favoring the intervention students. Statistically significant differences in aggression were reported by teachers, with those in the intervention group showing less aggression. There were marginally significant differences reported by parents, and no significant differences reported by observers. Students who participated in the intervention showed statistically significant improvements on direct assessments of the children's emotional understanding and social problem-solving skills and emergent literacy skills compared to the control group. When looking at language skills, those in the intervention group had more improvement on measures of child vocabulary and on parent reports of communication and language use at home. Observers also reported more improvements in task orientation among those in the intervention group compared to those in the control group.

Omnibus tests or overall tests of the significance grouping the individual measures in the conceptual domains were requested by Blueprints and subsequently delivered by the evaluators, finding significance in all but one (language skills) of the six domains (emergent literacy skills, social cognitions, social-emotional behaviors, learning engagement at school, and learning engagement at home).

Long-term: Bierman et al. (2014) examined the effects of the intervention on the 1-year follow-up measures (i.e., at the end of kindergarten). The intervention showed significant main effects on 5 of 13 outcome measures: phonemic decoding, teacher-rated learning behaviors, competent problem solving, teacher-rated aggression, and parent-rated aggression. Effect sizes ranged from .22 to .40. In most cases, these effect sizes were equivalent to or larger than those at posttest, and in most cases, they reflected benefits in the social-emotional domain.

Omnibus tests or overall tests of the significance grouping the individual measures in the conceptual domains were requested by Blueprints and subsequently delivered by the evaluators, finding significance in all but one (language/emergent literacy) of four domains (social competence, aggression, and learning engagement).

Significant moderation (p < .05) by kindergarten context emerged for 5 of 39 tests. The authors noted that, even in the absence of main effects, the interaction effects indicated program benefits for subsets of subjects. Specifically, the results indicated that the intervention had stronger effects and greater benefits when there were many low-achieving children in the school on four outcomes: teacher-rated attention problems, parent-rated attention problems, teacher-rated social competence, and teacher-rated aggression in schools. While these four interaction effects showed greater benefits in less positive kindergarten contexts, one other showed greater benefits in a positive kindergarten context. The results revealed stronger intervention effects on competent problem solving in classes with high-quality teacher-student interaction.

To isolate the changes unique to kindergarten, a final analysis added controls for posttest scores. Only one of the main effects of the intervention remained significant, and two were marginally significant. The weaker results suggest that most benefits came during preschool but that these gains were maintained in kindergarten. The moderating effects of kindergarten context, in contrast, were largely maintained with the controls for posttest scores.

Executive Function: In Bierman, Nix et al. (2008), the analysis examined the influence of measures of executive function on the skill and behavioral outcomes and the influence of the executive function measures as moderators of the intervention, as outcomes of the intervention, and as mediators of the intervention.

To determine if the program benefit varied by level of executive function, the models added product terms of treatment by each of the five executive function measures. Two of 35 interaction tests (five moderators by seven outcomes) reached statistical significance and three more reached marginal significance (p < .10). For these five interactions, children with low baseline levels of executive functioning benefitted more from the intervention; children with high baseline levels did well regardless of the intervention.

The intervention significantly improved (p < .05) posttest scores on one of the five executive function measures - Task Orientation - and came close (p = .06) on one more - DCCS. Effect sizes equaled .28 and .20, respectively, for the two outcomes.

Mediation tests examined the two executive function measures that were significantly or near significantly influenced by the program (Task Orientation and DCCS). Of the 14 tested mediation effects (two mediators by seven outcomes), three proved significant. Task Orientation significantly mediated the program effect on print awareness, with the relationship between the program and print awareness declining by 33% with controls for the two mediating variables. Task Orientation significantly mediated the program effect on observer-rated social competence and observer-rated aggression. The reductions due to the mediators equaled 29% for observer-rated social competence and 43% for observer-rated aggression.

In summary, 1 of 5 tests showed that executive function was significantly affected by the intervention, 2 of 35 tests showed that executive function moderated the intervention impact, and 3 of 14 tests showed that executive function mediated the intervention impact. Overall, 11% of the 54 tests involving the measures of executive function reached significance.

Developmental Trajectories: Tests of the latent class growth models showed program benefits for each of six outcomes tested over the five time points between posttest and 4-year follow-up: social competence, aggressive-oppositional behavior, learning behavior, attention problems, student-teacher closeness, and peer rejection. Compared to control children, intervention children had higher odds of belonging to positive trajectories and lower odds of belonging to negative trajectories.

Mediating Effects : The mediation analysis (Nix et al., 2013) treated changes in five outcome measures from baseline to posttest as mediators: 1) vocabulary, 2) emergent literacy skills, 3) emotional understanding, 4) competent social problem solving, and 5) positive social behavior. The intervention significantly improved all five of these mediating outcomes, with effect sizes ranging from .25 to .49.

The analysis examined three kindergarten outcomes: 1) reading achievement, 2) learning engagement, and 3) positive social behavior. The five mediators and three outcomes defined 15 possible indirect effects.

The results demonstrated that gains made during preschool predicted kindergarten functioning. For reading achievement, the intervention had significant indirect effects via four of the five mediators. For learning engagement, the intervention had significant indirect effects via three of five posttest measures. For positive social behavior, the intervention had significant indirect effects via one of five posttest measures (positive social behavior). Gains made during Head Start thus continued to predict functioning one year later, in kindergarten.

However, the study also found that the two conditions did not differ significantly on reading achievement and learning engagement in kindergarten. These results differ from Bierman et al. (2014), perhaps because Nix et al. (2013) used broader versions of the outcome measures.

6-year follow-up (Welsh et al., 2020)

The models did not show significant condition differences in growth over time from posttest to the follow-up but showed overall condition differences for three of the four outcomes. Over the six years following the intervention, children in the intervention condition, compared to the control condition, demonstrated significantly higher social adjustment (d = .34), academic engagement (d = .24), and parent involvement (d = .24). In summary, the results indicated that benefits of the intervention at posttest were maintained through the 6-year follow-up.

8- and 10-year follow-up (Bierman et al., 2020)

Across the 8- and 10-year follow-ups, children in the intervention condition, compared to the control condition, exhibited significantly fewer conduct problems (d = -.20) and emotional symptoms (d = -.25). Additional tests for clinical risk categories affirmed these effects.

Study 2

Summary

Bierman et al. (2015) evaluated the added benefit of home visits with REDI-P. This randomized trial involved 200 families randomly assigned to either the treatment of additional home visits or the control of the standard Head Start REDI program. The study recruited families from 24 Head Start centers in three urban and rural Pennsylvania counties during the 2008-2009 and 2009-2010 academic years. The study conducted assessments in the fall of prekindergarten and the end of kindergarten. Children completed standardized tests, teachers reported on student achievement and attitudes, parents and children were observed by researchers, and parents reported their own behavior and attitudes.

Bierman et al. (2015) found at posttest that the treatment group relative to the control group had significantly improved

• Standardized tests of language and literacy skills
• Emergent literacy
• Academic performance
• Self-directed learning
• Social competence
• Parent support for learning reading quality
• Parent support for learning in conversations

Evaluation Methodology

Design:

Recruitment: The study recruited families from 24 Head Start centers in three urban and rural Pennsylvania counties during the 2008-2009 and 2009-2010 academic years. Each year, the study sent letters describing the study to all prekindergarten children in participating classrooms. Letters were sent to approximately 400 eligible households, of which 52% elected to participate.

Assignment: The study randomly assigned 95 households to the treatment condition and 105 households to the control condition. Treatment families received the Head Start REDI materials in addition to home visits, while control conditions received only the Head Start REDI materials.

Attrition: The pretest assessment occurred in the fall of the preschool year and the posttest occurred in the spring of the kindergarten school year. The study reported that 4% of households left the study.

Sample: The sample was comprised of 55% white, 26% black, and 19% Hispanic children. The sample was 56% male and averaged 4.45 years of age. The median annual family income among the population was $18,000, 39% single parent, and 54% unemployed.

Measures: All but 2 of 11 measures came from independent observers - teachers and researchers who were unaware of condition. Two measures came from reports of parents, who helped deliver the program. Reliability was acceptable for most measures but was low (alpha = .56) for one of the parent-reported measures.

The study used four measures of child language and emergent literacy skills. First, the study used the Expressive One-Word Picture Vocabulary Test, which assessed vocabulary. Second, the study used the Letter-Word Identification scale of the Woodcock-Johnson Tests of Achievement III-revised, which tests emergent literacy skills. Third, the study used the Test of Word Reading Efficiency, which tests reading fluency. Finally, teachers, who were blind to condition, rated children's academic performance with the 12-item performance rating scale.

The study also used three teacher-rated measures to assess social-emotional adjustment. First, teachers used the School Readiness Questionnaire, which measured child self-directed learning. Second, teachers rated children on 13 items describing prosocial behavior. Finally, teachers rated children on the Teacher Observation of Child Adaptation-Revised, which describes aggressive behavior.

Finally, parent support for learning was assessed by four measures using parent reports, videotaped observations of parent-child interactions, and observations conducted in the home. Parents first described the degree to which they read interactively with their children, using five items from the Participation subscale of the Reading Belief Inventory and second described the quality of conversation with their children. Third, trained coders rated videotapes of parent-child interactions during structured tasks. Finally, observers rated parents with a modified version of the Post-Visit Inventory to measure parental warmth and support.

Analysis: The study used cross-classified hierarchical linear models, nesting children within their Head Start classrooms and elementary school districts. The models included pretest outcomes as covariates but the vocabulary measure served as the control for the other language and literacy measures that were obtained only in kindergarten. Other controls included kindergarten school quality, county, cohort, age, sex, race, aggression, single-parent status, caregiver depressive symptoms, and family income-to-needs ratio.

Intent-to-Treat: The study included all available data, excepting only the reported 4% attrition.

Outcomes

Implementation Fidelity: The study reported that families completed an average of 12 of 16 total visits, and visitors reported a parent-engagement score of 2.27 out of a possible 3.

Baseline Equivalence: The study reported that one outcome measure (parent support for learning) favored the comparison group at pretest, but all other sociodemographic and outcome measures were equivalent across the intervention and control groups at pretest.

Differential Attrition: With less than 4% attrition, the study did not provide additional analysis of attritors.

Posttest: A significant intervention effect was found for 2 of 4 measures of language and literacy skills. Children in the treatment group had significantly better scores in emergent literacy and academic performance at posttest as compared to children in the control group. In addition, the study found a significant intervention effect for 2 of 3 measures of social-emotional learning. Children in the treatment group had better scores at posttest in teacher-rated self-directed learning and social competence.

Finally, the study found a significant intervention effect for 2 of 4 risk and protective factors in parent support for learning. Parents in the treatment group reported higher scores for reading quality and conversations. Observer-rated scores on parent support did not differ across conditions.

Long-Term: The study did not report a long-term follow-up.

Study 3

Summary

Bierman et al. (2017) combined the two samples from Study 1 and Study 2 in a randomized controlled trial of 556 Head Start children assigned to three conditions: usual practice Head Start (control), REDI-C school-only intervention, and REDI-C plus the parent intervention. Assessments of social relations and academic outcomes in second grade occurred more than two years after the program end.

Bierman et al. (2017) found that in second grade, children in the REDI-C school-only program, compared to children in the control group, had significantly

• Higher teacher-rated social competence
• Higher teacher-rated classroom participation
• Higher student-teacher relationship quality
• Fewer teacher-rated peer problems.

Children in the school and parent program, compared to children in the REDI-C school-only program, showed significantly

• Higher student-reported social competence
• Fewer student-reported peer problems
• Better academic outcomes (sight word reading, reading skills, math skills).

Evaluation Methodology

Design:

Recruitment: The combined sample of two previous studies included 556 children who were recruited from Head Start centers in three counties in Pennsylvania. The children were 4-5 years old and eligible to start kindergarten in the following year. The participating Head Start centers came from one large county with both an urban community and smaller surrounding communities and from two smaller counties with small towns and rural areas.

While most eligible children (n=356; 86%) participated in the first study of the school program, only 52% (n=264) of the eligible parents agreed to participate in the second study of the school and parent program. The two samples thus differed, and as noted below, the authors weighted cases to control for selection into the second sample. Still, the representativeness of the two samples may differ.

Assignment: In the first study, 356 families were cluster randomized by Head Start center to the Head Start with REDI-C school intervention (n = 192) or the usual practice Head Start control group (n = 164). The randomization stratified the 25 Head Start centers on location, length of program (e.g., half day vs. full day), and student demographic characteristics (e.g., percent underrepresented minority).

In the second study, 200 families were individually randomized within classrooms by child to the combined REDI-C school and parent intervention (n = 95) or the school REDI-C comparison group (n = 105). The school-only parents received home materials via mail but in a different domain. Because all children participated in the school program, randomization of children could occur within classrooms and centers.

The full sample used in Study 3 thus consisted of three groups: usual practice Head Start (control), REDI-C school intervention only, and REDI-C plus the parent intervention. Note that the condition sample sizes listed in the CONSORT diagram do not match those listed in the text (p. 131). For example, the CONSORT diagram shows 95 in the school and parent program, while the text reports 105.

Assessments/Attrition: The follow-up assessment occurred near the end of second grade, 3.5 years after baseline (the start of the prekindergarten year) and more than two years after the program end. According to the CONSORT diagram, completion rates were 89% in sample 1 and 83% in sample 2.

Sample:

The combined sample averaged 4.48 years of age at baseline, had 49% males, and was 55% European American, 25% African American, and 19% Latino. Families were low-income (median annual income of $16,800), and 38% were single parents.

Measures:

The outcome measures came from researchers, students, and teachers. Trained research assistants visited schools, distributed rating forms to teachers, and conducted standardized child assessments during individual pull-out sessions. The research assistants who conducted the assessments and the second-grade teachers who provided ratings were unaware of the condition membership. The measures used previously validated instruments, and the scales had good reliabilities for the study sample.

The study divided the behavioral outcomes into two domains. In the domain labeled mental health, there were seven outcomes related to prosocial relationships with peers (e.g., social competence, peer problems) and risk and protective factors (e.g., classroom participation, student-teacher relationship quality). In the academic domain, there were five outcomes related to academic performance (e.g., letter-word identification, math skills).

Analysis:

The analysis used regression models with covariates for baseline outcomes, child and family demographic characteristics, preintervention child functioning, and study design features. The models presented separate tests for condition differences across the two samples, with the first tests comparing the school program to the control group, and the second tests comparing the combined school and parent program to the school program alone.

The analysis combined the two samples. Because consent differed widely between the two samples, the analysis weighted the second unrepresentative sample by propensity scores to match the characteristics of the first representative sample. The propensity scores came from a logistic regression using 44 covariates from the initial baseline assessment. After weighting, the two samples were equivalent within 0.2 standard deviations on all baseline measures.

Intent-to-Treat: The authors briefly noted that "Multiple imputation was used to estimate missing data" and that the intent-to-treat analysis included all families as randomized.

Outcomes

Implementation Fidelity:

Parents received (on average) 12 of the 16 planned home visits, and only a small number of families (13%) participated minimally, accepting 0-3 home visits.

Baseline Equivalence:

The study did not test for baseline equivalence, but information from each of the original and separate studies applies. The first sample (Study 1 above) showed no significant condition differences for the outcomes but did not examine sociodemographic measures. The second sample (Study 2) above showed one significant condition difference for the outcomes and sociodemographic measures (parent support for learning favored the control group.

Differential Attrition:

The study did not test for differential attrition, although attrition rates varied by condition. In sample 1, the attrition rate of 16% in the intervention group exceeded the attrition rate of 6% in the control group. In sample 2, the attrition rates were similar, 18% in the school-only condition and 16% in the school and parent condition. Most dropouts (73%) had moved or were unavailable, while the other 27% refused.

Posttest:

Not examined.

Long-Term:

In comparing the school program to the usual practice, five of seven tests of social relations outcomes were significant. The school intervention group did significantly better than the control group on teacher-reported measures of classroom participation, social competence, teacher-student relations, and peer problems and on student-reported perceived social competence. Effect sizes ranged from .24 to .39. However, none of the five academic outcomes differed significantly across conditions.

In comparing the combined school and parent program to the school-only program, two of seven tests of social relations outcomes were significant. The combined intervention group reported greater student-reported outcomes of perceived social competence (d = .43) and lower perceived peer problems (d = -.47). The combined intervention also did better for academic outcomes, significantly improving three of five outcomes: sight word reading, reading skills, and math skills. Effect sizes ranged from .30 to .50.

Figures 2 and 3 added the effect sizes from both samples, which ranged from .23 to .78.